Music Emotion Recognition via End-to-End Multimodal Neural Networks

Music emotion recognition (MER) is a key issue in user contextaware recommendation.Many existingmethods require hand-crafted features on audio and lyrics. Here we propose a new end-to-end method for recognizing emotions of tracks from their acoustic signals and lyrics via multimodal deep neural networks. We evaluate our method on about 7,000 K-pop tracks labeled as positive or negative emotion. The proposed method is compared to end-to-end unimodal models using audio signals or lyrics only. The experimental results show that our multimodal model achieves the best accuracy as 80%, and we discuss the reasons of these results.